fix(gooddata-eval): keep alert sim-user from accepting trigger drift by myhoai · Pull Request #1698 · gooddata/gooddata-python-sdk

myhoai · 2026-07-30T10:48:39Z

Summary

agent_alert_skill_2 and _15 fail on the staging nightly with the agent creating
ONCE_PER_INTERVAL/DAY alerts where the fixture expects ALWAYS, plus date filters nobody
asked for. Not a chatbot bug — the agent follows what the simulated user agrees to, and the
sim-user prompt in core/agentic/alert_skill.py had three gaps.

Verified still live: run agent_alert_skill_2026-07-29_03-11-11_gpt52_openai, item _15
(06756c61, "Alert me when our discounts are getting out of hand", expected Every time)
scored trigger_correct=0, with actual args trigger=ONCE_PER_INTERVAL,
trigger_interval=DAY, filters=[relativeDateFilter(order_date, DAY, -1..-1)].

1. The trigger was stated only when it differed from ALWAYS — so exactly the ALWAYS cases
went unsaid. This matters more than it looks: alert_prompt.md.jinja:333 instructs the agent to
use the date granularity as trigger_interval "unless the user explicitly chooses a different
interval", so an ALWAYS expectation only survives when the sim-user asks for it out loud. Now
stated for every explicit trigger, and still silent when the fixture omits Trigger (those cases
exercise the product default, which is ALWAYS — prepare_metric_alert_proposal_tool.py:389)
and for ANOMALY, whose product default really is ONCE_PER_INTERVAL and whose trigger
_check_trigger skips anyway.

2. The final-summary rule checked recipients only — a summary showing a wrong trigger got
confirmed. This is a regression: gdc-nas 2b4739d091 (QA-28113) already fixed it to check
recipients and trigger, and the GDAI-1830 thin-shim migration into this SDK dropped it.
Restored, and extended to filters.

3. Nothing forbade extra filters. The agent volunteers a date filter for a vague question and
then aligns the trigger interval to it, so refusing the filter removes the reason it had to change
the trigger. Fixing the trigger rule alone would be fragile against the product prompt.

Also: sampling temperature 0.5 → 0, since the drift was sampling-dependent.

Langfuse observability — each score now carries expected-vs-actual detail, e.g.
trigger_correct: expected 'ALWAYS'; actual 'ONCE_PER_INTERVAL/DAY'. These comments existed
before the same migration and were lost with it. Without them a failed check shows only 0.0,
and the create_metric_alert arguments have to be dug out of CI logs or of nested observations
on a different trace of the session than the one the scores land on (the scored trace is picked
by max latency, which for _15 was turn 2 — not the turn that created the alert).

Test Plan

Extended tests/test_agentic_alert_skill.py with 13 tests: trigger instruction per trigger
kind and its two silent cases, the no-extra-filter rule in both branches, rule numbering with
no gaps, temperature == 0, score-comment content incl. BETWEEN bounds and the
never-created case, and one covering the Langfuse branch end to end.
pytest tests/ → 256 passed (243 pre-existing + 13 new).
ruff check / ruff format --check clean on both changed files.
ty check → 15 diagnostics before and after, i.e. no new type errors.

Not yet verified live: _2/_15 repeat runs against staging on gpt52_openai, to confirm the
sim-user corrects at the summary and that the acceptance criteria hold end to end. Will be added
to this PR before merge.

Note for reviewers: one failure mode is out of scope here. The 07-29 20:22 gpt52_openai
run failed both items with alert_created=0 (the agent never called create_metric_alert).
This change does not address that, so the nightly can still go red for that reason.

Follow-up (deliberately not in this PR)

The GDAI-1830 migration dropped more than the three gaps above; all still missing:
threshold_from/threshold_to never reach the sim-user (so the BETWEEN item runs with
threshold=not specified), the ANOMALY insistence rule, _describe_filters_for_prompt, and the
local order-sensitive _deep_subset duplicating the order-insensitive
core/evaluators/_deep_subset.deep_subset. Kept out to keep this diff scoped to the ticket.

JIRA: https://gooddata.atlassian.net/browse/QA-28623
Risk: nonprod — test-infrastructure only; no product or library code paths change.

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved alert evaluation for unspecified triggers, recipient matching, filters, and threshold ranges.
- Made simulated alert behavior deterministic and more consistent.
- Enhanced evaluation details with clearer scoring comments, including trigger, filter, recipient, and threshold information.
Tests
- Added coverage for trigger semantics, filter constraints, prompt formatting, deterministic behavior, and evaluation trace comments.

The staging nightly repeatedly failed agent_alert_skill_2 and _15: the agent created ONCE_PER_INTERVAL/DAY alerts where the fixture expects ALWAYS, plus date filters nobody asked for. The chatbot was following the simulated user, whose prompt had three gaps: - The expected trigger was stated only when it differed from ALWAYS, so exactly the ALWAYS cases went unsaid. The alert prompt tells the agent to align trigger_interval with any date granularity in play "unless the user explicitly chooses a different interval", so an ALWAYS expectation only survives when the sim-user asks for it out loud. Now stated for every explicit trigger, and still silent both when the fixture omits Trigger (those cases exercise the product default) and for ANOMALY, whose product default really is ONCE_PER_INTERVAL. - The final-summary verification rule checked recipients only, so a summary showing a wrong trigger got confirmed anyway. It checked the trigger too until the logic moved into this SDK; restored, and extended to filters. - Nothing forbade extra filters. The agent volunteers a date filter for a vague question and then aligns the trigger interval to it, so refusing the filter removes the reason it had to change the trigger in the first place. Fixing the trigger rule alone would be fragile. Sampling temperature drops to 0 as well, since the drift was sampling-dependent. Each Langfuse score now carries expected-vs-actual detail. These comments also existed before the same migration and were lost with it: without them a failed trigger/filters check shows only 0.0 on the trace, and the create_metric_alert arguments have to be dug out of CI logs or of nested observations on a different trace of the session than the one the scores land on. JIRA: QA-28623 risk: nonprod

coderabbitai · 2026-07-30T10:48:55Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 531c27b8-5efe-49f3-8044-2529a9209229

📥 Commits

Reviewing files that changed from the base of the PR and between 0382d27 and dbbe6c7.

📒 Files selected for processing (2)

packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py
packages/gooddata-eval/tests/test_agentic_alert_skill.py

📝 Walkthrough

Walkthrough

The alert-skill evaluator now builds deterministic, rule-based simulated-user prompts, normalizes recipient arguments, handles unspecified triggers, and reports detailed per-check comments with Langfuse scores. Tests cover prompt rules, threshold formatting, scoring comments, and trace integration.

Changes

Alert skill evaluation

Layer / File(s)	Summary
Prompt rules and alert validation `packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py`, `packages/gooddata-eval/tests/test_agentic_alert_skill.py`	Trigger and filter instructions are assembled as numbered rules, unspecified triggers remain silent, recipient arguments are normalized for set comparison, threshold operators share a constant, and OpenAI sampling uses temperature `0`. Tests cover these prompt and validation behaviors.
Scoring comment generation and Langfuse reporting `packages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.py`, `packages/gooddata-eval/tests/test_agentic_alert_skill.py`	Per-check comments now include trigger, threshold, filter, metric, and recipient details, including the alert-not-created case, and are attached to Langfuse scores. Tests verify comment formatting and trace output.
Evaluation trace integration coverage `packages/gooddata-eval/tests/test_agentic_alert_skill.py`	The Langfuse evaluation path is exercised with mocked tool-call events and checked for trigger and filter score comments.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Evaluator
  participant AlertToolCall
  participant LangfuseTrace
  Evaluator->>AlertToolCall: read actual alert arguments
  Evaluator->>Evaluator: generate per-check scoring comments
  Evaluator->>LangfuseTrace: submit boolean scores and comments

Suggested reviewers: hkad98, lupko, pcerny

Poem

A bunny checks each rule in line,
With triggers clear and numbers fine.
Recipients hop into a set,
Langfuse gets the notes they get.
“Temperature zero!” the rabbit sings,
While comments sprout on scoring strings.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: preventing the simulated alert user from accepting trigger drift.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

_{Comment @coderabbitai help to get the list of available commands.}

myhoai requested review from hkad98, lupko and pcerny as code owners July 30, 2026 10:48

myhoai closed this Jul 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(gooddata-eval): keep alert sim-user from accepting trigger drift - #1698

fix(gooddata-eval): keep alert sim-user from accepting trigger drift#1698
myhoai wants to merge 1 commit into
masterfrom
QA-28623-alert-sim-user-drift

myhoai commented Jul 30, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

myhoai commented Jul 30, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Follow-up (deliberately not in this PR)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

myhoai commented Jul 30, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading